Skip to content

[HBASE-26534] Update HBase version to 2.4, make Hadoop 3 and Spark 3 default versions.#88

Merged
taklwu merged 4 commits intoapache:masterfrom
LucaCanali:updateToHBase24
Jan 13, 2022
Merged

[HBASE-26534] Update HBase version to 2.4, make Hadoop 3 and Spark 3 default versions.#88
taklwu merged 4 commits intoapache:masterfrom
LucaCanali:updateToHBase24

Conversation

@LucaCanali
Copy link
Contributor

@LucaCanali LucaCanali commented Dec 2, 2021 •
edited
Loading

This proposes to bump up the default versions used by HBase-connectors to recent versions, in particular for HBase, Spark, and Hadoop:
HBase is upgraded to 2.4.9, Spark to 3.1.2, Hadoop to 3.2.0.
This proposes also to make Spark 3 and Hadoop 3 default version (as opposed to Spark 2 and Hadoop 2).
Additional minor changes to the code are proposed/needed to compile the connector with HBase 2.4.9.

@Apache-HBase
Copy link

Apache-HBase commented Dec 2, 2021

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 54s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 spotbugs 0m 0s spotbugs executables are not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ master Compile Tests _
+0 🆗 mvndep 0m 23s Maven dependency ordering for branch
+1 💚 mvninstall 1m 13s master passed
+1 💚 compile 0m 47s master passed
+1 💚 javadoc 0m 48s master passed
+1 💚 scaladoc 0m 0s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 19s Maven dependency ordering for patch
+1 💚 mvninstall 0m 50s the patch passed
+1 💚 compile 0m 46s the patch passed
+1 💚 javac 0m 46s the patch passed
+1 💚 scalac 0m 46s the patch passed
-1 ❌ whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 💚 xml 0m 2s The patch has no ill-formed XML file.
+1 💚 javadoc 0m 46s the patch passed
+1 💚 scaladoc 0m 0s the patch passed
_ Other Tests _
+1 💚 unit 7m 12s root in the patch passed.
14m 58s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/1/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR #88
Optional Tests dupname javac javadoc unit spotbugs findbugs compile xml markdownlint scalac scaladoc
uname Linux b4309709c03d 5.4.0-1054-aws #57~18.04.1-Ubuntu SMP Thu Jul 15 03:21:36 UTC 2021 x86_64 GNU/Linux
Build tool hb_maven
Personality dev-support/jenkins/hbase-personality.sh
git revision master / 9a839b2
Default Java Oracle Corporation-1.8.0_282-b08
whitespace https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/1/artifact/yetus-precommit-check/output/whitespace-eol.txt
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/1/testReport/
Max. process+thread count 892 (vs. ulimit of 12500)
modules C: kafka/hbase-kafka-proxy spark spark/hbase-spark . U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/1/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented Dec 2, 2021

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 57s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 spotbugs 0m 0s spotbugs executables are not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ master Compile Tests _
+0 🆗 mvndep 0m 23s Maven dependency ordering for branch
+1 💚 mvninstall 1m 28s master passed
+1 💚 compile 0m 51s master passed
+1 💚 javadoc 0m 50s master passed
+1 💚 scaladoc 0m 0s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 20s Maven dependency ordering for patch
+1 💚 mvninstall 0m 52s the patch passed
+1 💚 compile 0m 49s the patch passed
+1 💚 javac 0m 49s the patch passed
+1 💚 scalac 0m 49s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 2s The patch has no ill-formed XML file.
+1 💚 javadoc 0m 48s the patch passed
+1 💚 scaladoc 0m 0s the patch passed
_ Other Tests _
+1 💚 unit 7m 24s root in the patch passed.
15m 41s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/2/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR #88
Optional Tests dupname javac javadoc unit spotbugs findbugs compile xml markdownlint scalac scaladoc
uname Linux f4ba024dd4d1 5.4.0-1047-aws #49~18.04.1-Ubuntu SMP Wed Apr 28 23:08:58 UTC 2021 x86_64 GNU/Linux
Build tool hb_maven
Personality dev-support/jenkins/hbase-personality.sh
git revision master / 9a839b2
Default Java Oracle Corporation-1.8.0_282-b08
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/2/testReport/
Max. process+thread count 916 (vs. ulimit of 12500)
modules C: kafka/hbase-kafka-proxy spark spark/hbase-spark . U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/2/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ianawilson added a commit to sproutsocial/hbase-connectors that referenced this pull request Dec 27, 2021
@ianawilson
Merge updateToHBase24 from LucaCanali: apache#88
45971a1 
The purpose of this is to match our hbase-server version with the jar
provided on EMR. While our HBase is listed as 2.2.6, the library code we
have from Amazon is for 2.4.1+. There was a breaking change in 2.4.1
and this mismatch prevents bulkLoadThinRows() from working.
@ianawilson
Copy link

ianawilson commented Dec 28, 2021

This is great. I ran into an issue with running this on AWS EMR and which provides HBase 2.4.1+ library code, and this had exactly the updates I needed.

I'd love to see this merged and packaged in a release.

@ianawilson
Copy link

ianawilson commented Dec 28, 2021

One thing to think about with this:

HBase 2.4.1 introduced (via this PR apache/hbase#2800) a breaking change for bulkLoadThinRows() by moving some of the static helpers from HStore to StoreUtils (this was precisely the problem that lead me to this PR).

This PR enables you to use this with HBase 2.4.1 and later, but will prevent you from using it with earlier HBase versions. I know you currently allow the HBase version to be configured via the system properties, but as-is this change won't work with the HBase library prior to 2.4.1.

@ianawilson ianawilson mentioned this pull request Dec 29, 2021
Closed
@LucaCanali
Copy link
Contributor Author

LucaCanali commented Jan 11, 2022

Thanks @ianawilson for looking at this.
My thought and proposal with this is that if HBase 2.4.1 could afford a breaking change, that could reflect on the HBase connector too. Indeed this PR this requires HBase client 2.4.x to work. However, this is only for the HBase client version, it does not affect the server version, as many people are still using HBase versions older than 2.4.

ianawilson
ianawilson approved these changes Jan 12, 2022
View reviewed changes
Copy link

@ianawilson ianawilson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LucaCanali Makes sense to me!

I'm not a contributor to this repo, so I don't know how much weight my approval carries, but I've been using this branch successfully in a couple of ad hoc Spark jobs with HBase client 2.4, and it's been great.

@ianawilson
Copy link

ianawilson commented Jan 12, 2022

@LucaCanali FYI, looks like there's a PR with a similar intent and slightly different approach #89

@taklwu
Copy link
Contributor

taklwu commented Jan 12, 2022 •
edited
Loading

@LucaCanali do you mind merging my changes from #89 to here? I will review yours instead, and that could help to move upgrade the forward .

taklwu
taklwu reviewed Jan 12, 2022
View reviewed changes
Copy link
Contributor

@taklwu taklwu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you run the unit test on your local environment ? may I ask if you're using linux not mac?

basically, I had some changes in TestJavaHBaseContext.java and other *Suite.scala , please see them in my PR #89 , that helps to prevent unit test failure on my mac. but it may be only me having these issues. if you can, please help to merge those changes in unit tests as well and IMO it's helping to run it smooth without conflicts between each tests.

@LucaCanali LucaCanali changed the title [HBASE-26534] Update HBase version to 2.4.8, make Hadoop 3 and Spark 3 default versions. [HBASE-26534] Update HBase version to 2.4, make Hadoop 3 and Spark 3 default versions. Jan 13, 2022
@Apache-HBase
Copy link

Apache-HBase commented Jan 13, 2022

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 15s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 spotbugs 0m 1s spotbugs executables are not available.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ master Compile Tests _
+0 🆗 mvndep 0m 23s Maven dependency ordering for branch
+1 💚 mvninstall 2m 55s master passed
+1 💚 compile 0m 50s master passed
+1 💚 javadoc 0m 48s master passed
+1 💚 scaladoc 0m 0s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 21s Maven dependency ordering for patch
+1 💚 mvninstall 0m 53s the patch passed
+1 💚 compile 0m 48s the patch passed
+1 💚 javac 0m 48s the patch passed
+1 💚 scalac 0m 48s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 2s The patch has no ill-formed XML file.
+1 💚 javadoc 0m 48s the patch passed
+1 💚 scaladoc 0m 0s the patch passed
_ Other Tests _
+1 💚 unit 7m 18s root in the patch passed.
17m 14s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/3/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR #88
Optional Tests dupname javac javadoc unit spotbugs findbugs compile xml markdownlint scalac scaladoc
uname Linux 1b6d27673a83 5.4.0-1047-aws #49~18.04.1-Ubuntu SMP Wed Apr 28 23:08:58 UTC 2021 x86_64 GNU/Linux
Build tool hb_maven
Personality dev-support/jenkins/hbase-personality.sh
git revision master / 9a839b2
Default Java Oracle Corporation-1.8.0_282-b08
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/3/testReport/
Max. process+thread count 917 (vs. ulimit of 12500)
modules C: kafka/hbase-kafka-proxy spark spark/hbase-spark . U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/3/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@LucaCanali
Copy link
Contributor Author

LucaCanali commented Jan 13, 2022

Thanks @taklwu for comments and review.
I run my tests on Linux and they are all OK, however I have added your proposed changes to TestJavaHBaseContext.java too.

@Apache-HBase
Copy link

Apache-HBase commented Jan 13, 2022

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 44s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 spotbugs 0m 0s spotbugs executables are not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ master Compile Tests _
+0 🆗 mvndep 0m 28s Maven dependency ordering for branch
+1 💚 mvninstall 2m 47s master passed
+1 💚 compile 0m 48s master passed
+1 💚 javadoc 0m 48s master passed
+1 💚 scaladoc 0m 0s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 25s Maven dependency ordering for patch
+1 💚 mvninstall 0m 48s the patch passed
+1 💚 compile 0m 47s the patch passed
+1 💚 javac 0m 47s the patch passed
+1 💚 scalac 0m 47s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 3s The patch has no ill-formed XML file.
+1 💚 javadoc 0m 45s the patch passed
+1 💚 scaladoc 0m 0s the patch passed
_ Other Tests _
+1 💚 unit 7m 37s root in the patch passed.
16m 59s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/4/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR #88
Optional Tests dupname javac javadoc unit spotbugs findbugs compile xml markdownlint scalac scaladoc
uname Linux 46e8c32c7378 5.4.0-1054-aws #57~18.04.1-Ubuntu SMP Thu Jul 15 03:21:36 UTC 2021 x86_64 GNU/Linux
Build tool hb_maven
Personality dev-support/jenkins/hbase-personality.sh
git revision master / 9a839b2
Default Java Oracle Corporation-1.8.0_282-b08
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/4/testReport/
Max. process+thread count 903 (vs. ulimit of 12500)
modules C: kafka/hbase-kafka-proxy spark spark/hbase-spark . U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-88/4/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

taklwu
taklwu approved these changes Jan 13, 2022
View reviewed changes
Copy link
Contributor

@taklwu taklwu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM ! and thanks for merging the changes

@taklwu taklwu merged commit 7578a33 into apache:master Jan 13, 2022
@taklwu taklwu mentioned this pull request Jan 13, 2022
Closed
@ianawilson
Copy link

ianawilson commented Jan 13, 2022

This is great, thank you both @LucaCanali @taklwu! One question I've got: Is it possible for this update to be released and published to maven central? It seems like there are quite a few updates on this repository that haven't been released there yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@taklwu taklwu taklwu approved these changes

@petersomogyi petersomogyi Awaiting requested review from petersomogyi

@meszibalu meszibalu Awaiting requested review from meszibalu

+1 more reviewer

@ianawilson ianawilson ianawilson approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@LucaCanali @Apache-HBase @ianawilson @taklwu